Complex geocoding config [MAP-706] [MAP-708] [MAP-707] [MAP-458] #166

janbaykara · 2024-12-17T12:32:24Z

This PR adds a new, more capable geocoding option that can involve multiple fields.

geography_config is a new approach to geocoding. When the field is set (see test cases), it allows three new approaches to geocoding:
1. Multi-field area geocoding. It can be a single area name, or it could be multiple columns (like council -> ward). A good example of this is the Britain Elects ward-level GE2024 results forecast.
2. Address geocoding with per-row prefix / suffix / country configurations. A good example of this is a spreadsheet with business names and street names. Both fields are important to get precise coordinates, and adding an arbitrary suffix like "Glasgow" gives the geocoder the context it needs to make the right choices.
3. Coordinate geocoding. You pick a latitude and longitude field, and the rest is history. This was an explicit request from Tipping Point UK.
There is a full test suite for these geocoding configurations, and they run on the Github CI test runner as a follow-up step.
As part of area geocoding, we now import inactive political boundaries from MapIt, taking the row count from 9000~ to over 20,000.
There are a few cases where we use a new service, findthatpostcode.uk, for backup postcode geocoding when postcodes.io fails.
When geography_config is set, geocoding cannot be set differently by the user in the UI. So there are some JSX changes for that.

…th GSS codes

x

sentry-io · 2024-12-17T12:32:46Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: hub/management/commands/import_areas.py

Function	Unhandled Issue
`handle`	BadRequestException: Usage limit reached. utils.m... `Event Count:` 1

_{Did you find this useful? React with a 👍 or 👎}

joaquimds

Looks good to me, I just had a few minor comments.

hub/data_imports/geocoding_config.py

joaquimds · 2025-01-07T13:40:14Z

hub/models.py

        if any([job.status == "doing" for job in jobs]):
            status = "doing"
        elif any([job.status == "failed" for job in jobs]):
            status = "failed"
        elif all([job.status == "succeeded" for job in jobs]):
            status = "succeeded"
+        elif number_of_jobs_ahead_in_queue <= 0:
+            status = "succeeded"


question: what's the logic here?

The GraphQL query was responding with status succeeded even if you had cancelled some of the jobs. So I added this logic: if there are no more jobs, report that the status is done. There wasn't a done status, so I settled for succeeded because I'm an optimist starting 2025.

joaquimds · 2025-01-07T13:41:52Z

hub/models.py

@@ -1836,7 +1841,7 @@ async def update_many(self, mapped_records: list[MappedMember], **kwargs):
            "Update many not implemented for this data source type."
        )

-    def get_record_id(self, record):
+    def get_record_id(self, record) -> Optional[Union[str, int]]:


question: lots of things break if get_record_id returns None. Can we make this non-optional? if so, what do we return instead of None?

Can we return a false ID without that causing genuine problems? That seems like a dodgy answer to me. Instead I think we should handle the case.

The case here is that I added a Google Sheet with an empty ID cell, so Django did return a None. I didn't change the behaviour, I just specified typing that reflected reality.

You might be correct but I think that this is a separate ticket to solve the problem of "what happens when data sources return a None ID?" Because this is already a live problem in the Mapped system.

We could do some heuristic work to generate our own IDs, but then what if the ID in the system changes? Then suddenly we'll have duplicate data in Mapped.

I think the best approach is to keep get_record_id honest and then deal with None IDs with the principle that we can't use them, basically. New ticket for this: https://linear.app/commonknowledge/issue/MAP-767/gracefully-handle-reject-records-with-a-none-id

joaquimds · 2025-01-07T13:43:52Z

hub/tests/fixtures/geocoding_cases.py

+        "ward": "Credenhill",
+        "expected_area_type_code": "WD23",
+        "expected_area_gss": "E05012957",
+    },


suggestion: have a "X city council" vs "X council" case here

Not sure I follow, can you expand?

hub/tests/fixtures/geocoding_cases.py

…ard-csv-issues

janbaykara added 19 commits December 10, 2024 17:15

Multi-column geogoding script to enrich Britain Elects spreadsheet wi…

1454517

…th GSS codes

Filter council names correctly

fc1d637

Better matching

32d3667

Start at gen 10

5299521

Ignore areas without a GSS code

42b671d

Edge case: remap "Bristol, city of" to "bristol city" for mapit matching

bbdace2

Remove logging

89047e2

Pass in update data

e4777aa

Merge branch 'main' into feature/map-668-hnh-ward-csv-issues

ff72ea6

Rename and reorganise tests for external data source

ca242e9

x

Tweaks

a60ddd1

Prepare loaders even if EDS has no org

b222cb2

Ingest area data even if multi-level geocoding fails

deb4265

Prepared tests for tricky multi-level geocoding cases

55f62f4

Skip geocoding test while areas aren't in the DB

afabd19

Add geocoding data for tracing purposes

bea8cb4

x

Tweaks

0bd0119

Fix tests, use seed data

86f9fd3

Make coordinates work

22b6897

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-database PR #166 December 17, 2024 12:32 — with Render Active

janbaykara temporarily deployed to testing December 17, 2024 12:32 — with GitHub Actions Inactive

janbaykara had a problem deploying to testing December 17, 2024 12:32 — with GitHub Actions Failure

janbaykara changed the title ~~Feature/map 668 hnh ward csv issues~~ Complex geocoding config Dec 17, 2024

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 December 17, 2024 12:33 — with Render Destroyed

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 December 17, 2024 12:33 — with Render Destroyed

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 December 17, 2024 12:33 — with Render Active

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 December 17, 2024 12:33 — with Render View deployment

commonknowledge-bot had a problem deploying to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 December 17, 2024 12:33 — with Render Failure

commonknowledge-bot had a problem deploying to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 6, 2025 16:31 — with Render Failure

janbaykara had a problem deploying to testing January 6, 2025 16:31 — with GitHub Actions Failure

commonknowledge-bot had a problem deploying to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 January 6, 2025 16:31 — with Render Failure

joaquimds reviewed Jan 7, 2025

View reviewed changes

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 January 7, 2025 14:13 — with Render Active

commonknowledge-bot had a problem deploying to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 January 7, 2025 14:13 — with Render Failure

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 7, 2025 14:13 — with Render View deployment

janbaykara added 3 commits January 7, 2025 17:11

Remove irrelevant comments

46c195e

Remove unnecessary case change

897253a

Allow children to be passed as props in React.

d9c4d94

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 7, 2025 17:18 — with Render Destroyed

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 January 7, 2025 17:19 — with Render Destroyed

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 January 7, 2025 17:19 — with Render Destroyed

Merge remote-tracking branch 'origin/main' into feature/map-668-hnh-w…

d9cd7e8

…ard-csv-issues

janbaykara had a problem deploying to testing January 7, 2025 17:19 — with GitHub Actions Failure

janbaykara temporarily deployed to testing January 7, 2025 17:19 — with GitHub Actions Inactive

commonknowledge-bot requested a deployment to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 January 7, 2025 17:19 — with Render In progress

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-frontend PR #166 January 7, 2025 17:19 — with Render Destroyed

commonknowledge-bot requested a deployment to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 January 7, 2025 17:19 — with Render In progress

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 January 7, 2025 17:19 — with Render Active

commonknowledge-bot requested a deployment to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 7, 2025 17:19 — with Render In progress

commonknowledge-bot deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 7, 2025 17:19 — with Render View deployment

janbaykara had a problem deploying to testing January 7, 2025 17:37 — with GitHub Actions Failure

fix: change test record email addresses to prevent stale data

4033fce

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-backend PR #166 January 7, 2025 17:58 — with Render Destroyed

joaquimds temporarily deployed to testing January 7, 2025 17:58 — with GitHub Actions Inactive

commonknowledge-bot temporarily deployed to feature/map-668-hnh-ward-csv-issues - meep-intelligence-hub-worker PR #166 January 7, 2025 17:58 — with Render Destroyed

janbaykara merged commit 4a3bf76 into main Jan 7, 2025
4 checks passed

janbaykara deleted the feature/map-668-hnh-ward-csv-issues branch January 7, 2025 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complex geocoding config [MAP-706] [MAP-708] [MAP-707] [MAP-458] #166

Complex geocoding config [MAP-706] [MAP-708] [MAP-707] [MAP-458] #166

janbaykara commented Dec 17, 2024 •

edited

Loading

sentry-io bot commented Dec 17, 2024

joaquimds left a comment

joaquimds Jan 7, 2025

janbaykara Jan 7, 2025 •

edited

Loading

joaquimds Jan 7, 2025

janbaykara Jan 7, 2025 •

edited

Loading

janbaykara Jan 7, 2025 •

edited

Loading

joaquimds Jan 7, 2025

janbaykara Jan 7, 2025

Complex geocoding config [MAP-706] [MAP-708] [MAP-707] [MAP-458] #166

Complex geocoding config [MAP-706] [MAP-708] [MAP-707] [MAP-458] #166

Conversation

janbaykara commented Dec 17, 2024 • edited Loading

sentry-io bot commented Dec 17, 2024

🔍 Existing Issues For Review

joaquimds left a comment

Choose a reason for hiding this comment

joaquimds Jan 7, 2025

Choose a reason for hiding this comment

janbaykara Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

joaquimds Jan 7, 2025

Choose a reason for hiding this comment

janbaykara Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

janbaykara Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

joaquimds Jan 7, 2025

Choose a reason for hiding this comment

janbaykara Jan 7, 2025

Choose a reason for hiding this comment

janbaykara commented Dec 17, 2024 •

edited

Loading

janbaykara Jan 7, 2025 •

edited

Loading

janbaykara Jan 7, 2025 •

edited

Loading

janbaykara Jan 7, 2025 •

edited

Loading